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Abstract 

This chapter characterizes human problem solving in digital circuit design. 
We analyze protocols of designers with varying degrees of training, identify 
problem solving strategies used by these designers, discuss activity patterns 
that differentiate designers, and propose these as a tentative basis for assessing 
expertise in digital design. Throughout, we argue that a comprehensive model 
of human design should integrate a variety of strategies, which heretofore have 
been proposed as individually sufficient models of human design problem solv- 
ing. We close by describing an automated tool for design and its assessment. 
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1 Overview 

Cognitive diagnosis of expertise relies on having a characterization of expertise in the 
domain. The focus of our work is on the design of digital circuits. In this domain, pre- 
vious work was limited either to analog circuits or to much simpler elements of circuit 
design than that involved in the complex circuits we were interested in. However, the 
design of complex circuits allowed greater possibilities for observing multiple levels of 
expertise as well as individual differences in design problem solving. In earlier work 
(James, Goldman, Vandermolen, 1993; Vandermolen, etc.), we had focused on the 
design of a simpler digital circuit and the need for a richer design problem became 
apparent. Our investigation of complex digital circuit design had two major goals: 
to identify characteristics of design problem solving in this domain, and to determine 
how to appropriately characterize and differentiate among individuals using different 
design problem solving processes. When designing complex circuits, expert designers 
attempt to catch flaws as early in the design process as possible. Hence, we were 
interested in characterizing the processes involved in producing the final design, as 
well as the completeness and correctness of the final design itself. 

We begin by describing a framework for ; nalyzing design processes. Our empiri- 
cal work uses this framework to analyze the design processes of subjects with varying 
levels of experience. However, from a cognitive diagnosis perspective, process assess- 
ment is resource intensive and it would be nice if classification of individuals could 
be solely based on the final product. We evaluate the relative information supplied 
by process and product characterizations for classification of expertise. We conclude 
that both process and product measures provide important information. We describe 
the design of an automated tool for collecting process and product information so as 
to make such assessment tractable on a wide scale. 

2 Introduction to Engineering System Design 

Engineering system design maps a specified function onto a realizable physical struc- 
ture [Tong and Sriram, 1992]. This typically requires that the functional specification 
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of a system be decomposed, components be identified that satisfy various subfunc- 
tions, and that these components by integrated in a way that satisfies the overall 
system specification. Brown and Chandrasekaran [1986] and Sriram [1986] classify 
design tasks as follows. 

1. Routine design is indicated when effective problem decompositions are known, 
the mapping from subfunctions to physical components is clear, and the task is 
to select the most appropriate set of components that optimize well-established 
criteria. 

2. Innovative design assumes that top-level functional decompositions are known, 
but the physical realizations of subfunctions requires considerably more effort, 
such as constructing a solution from scratch, or making substantial functional 
and structural modifications to a known solution. 

3. Creative design is called for when the functional specification is open-ended 
and/or ill-specified, effective decompositions are unknown, and the designer 
must evaluate a number of different options to determine an appropriate design 
plan. 

This was proposed as a taxonomy of design tasks, but we feel that it better 
describes design tasks conditional on abilities of a designer. For example, the de- 
sign of digital circuitry such as a network controller, is likely to be routine to a 
designer with considerable experience on such problems. In contrast, persons who 
are knowledgeable of circuit design and understand controller functionality, but have 
little experience designing such controllers are likely to take an innovative design 
approach. Functional decomposition will be easily achieved, but considerable effort 
will be expended in picking and composing suitable components to satisfy the over- 
all functionality. Lastly, subjects who understand the domain of digital circuits and 
systems but have no experience in performing complex design, may try a number of 
decompositions and build subfunctionalities incrementally by trial and error. This is 
best characterized as creative design. 

In sum, a routine task for an expert may well be a creative design task for a novice. 
We believe that characteristics of routine, innovative, and creative design process, as 
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well as characteristics of the final design, are important in assessing the expertise of 
designers. We distinguish three broad levels of design activity. 

1. Design activities at the function level study the specifications of a system and 
generate functional units that collectively meet the required specifications. 

2. The transition level includes design activities that look to functional units and 
decide how to implement them as artifacts (e.g., gates and higher level blocks 
such as shift registers and counters). Actions at this level may also guide the 
coordination of component artifacts so that higher-level functionality is satisfied. 

3. Implementation level activities deal with the actual generation of the artifact. In 
complex design problems this often happens in stages, for example, simulation, 
testing, and evaluation of partial designs. 

These levels are closely tied to our earlier classification of design tasks: routine 
design is revealed by ease in addressing subtasks at each of the function, transition, 
and implementation levels; innovative design is suggested by behavioral determin- 
ism by a designer at the function level, but nondeterminism at the transition and 
implementation levels; designers with difficulty at each level are, by virtue of their 
apparent abilities, performing creative design. The activities occurring at these levels 
constitute indices of design problem solving processes. The final result of these pro- 
cesses can be evaluated to determine whether the design meets the necessary circuit 
specifications. 

3 Expertise in Design 

The previous section distinguished function, transition, and implementation levels; 
these relate design activities to views or abstractions of the particular system being 
designed. Researchers have also described orthogonal g: "lients that characterize the 
cognitive processes of the designer, independent of the artifact being designed. For 
example, Korpi, Greeno, and Jackson [1991] model design problem solving with two 
spaces: (a) the problem construction space, and (b) the design construction space. The 
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problem construction space is populated by what they call senior managers who guide 
the design and do most of the high level planning. The design construction space has 
middle managers, who act as facilitators in deciding what part of the design to work 
on and also assess the goodness of an emerging solution, and builders who actually 
perform the detailed design tasks. Similarly, Goel and Pirolli [1991] look upon design 
activity in two parts: (a) the focus of the activity (monitoring, development, etc.), and 
(b) the problem solving step being executed, which includes a set of operators, such as 
add, modify, evaluate, propose, and request. Ullman et al. [1988] do not differentiate 
a small number of discrete levels of cognitive processing, but they nonetheless posit 
that designers manipulate behavioral episodes at varying levels of detail. 

In our work, we have found two descriptive levels of reasoning sufficient: the 
strategy and unit operator levels. This chapter focuses exclusively on activity at 
the strategic level, roughly corresponding to the high-level management activities of 
Korpi, Greeno, and Jackson (1991). This and other work suggests high-level problem- 
solving strategies that might populate this space. 

1. Designers often rely on a decomposition strategy that decomposes a complex 
design task into smaller subtasks until they are directly solvable. Examples of 
design systems that employ decomposition are Hi-Rise [Maher, 1988] and Rl 
[McDermat, 1982]. 

2. A transformation/refinement strategy converts initial specifications into a final 
design solution through a sequence of primitive operator applications in some 
formal representation scheme (e.g., shape grammars [Mitchell and Stiny, 1978]). 

3. A case-based strategy assumes that a catalog of previous design solutions is 
stored in memory; a new design problem prompts the designer to search through 
this catalog, and select one or more designs that best match the characteristics 
of the goal design. Examples of case-based approaches to design include the 
Bogart system [Mostow, Barley, and Weinrich, 1989] and Struple [Zhao and 
Maher, 1988]. 

Most automated design systems and studies of the human design process have 
tended to focus on one strategy. In contrast, Maher [1990] discusses the relevance of all 
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three strategies to design and states "the value in identifying multiple models of design 
lies in the richness of the representation of design knowledge and experience provided 
by each and in the ability to choose a model that more closely fits the knowledge readily 
available for the domain being considered. " Foi example, a designer may decompose a 
specific ation, access previously constructed components or cases that best match the 
subfunctions identified through decomposition, and transform these cases into new- 
component structures that perfectly fit the requirements of the new system being 
designed. Thus, all three strategies may be present in the solution of a single design 
problem. 

We hypothesize the presence of each of these strategies in designers. The possi- 
bility of multiple strategies in a single design solution provides a much richer frame- 
work for characterizing the reasoning methods that designers employ in complex 
design problem solving. We look upon design as an application of reasoning pro- 
cesses that start at the function level, go through a transition level, and produce an 
implementation-level description of the artifact. Several strategies can be applied in 
this process. The directedness with which these strategies are employed and coordi- 
nated is indicative of routine, innovative, or creative design, and we believe, can serve 
as a basis for assessing a designer's expertise relative to a design problem or class of 
problems. 

4 Complex Circuit Design 

Our objective is to determine whether designers employ multiple problem-solving 
strategies, and if so, what is the interaction of these strategies across levels of ex- 
pertise. We have selected circuit design as a testbed for these studies. In particular, 
consider the problem of designing a network controller, which coordinates communi- 
cations between computers that are linked by cable. A more complete specification 
is given in Appendix A. Our experience with this and similar problems suggested 
numerous activities that would be observed across a range of designers. We classify 
these activities into the function, transition, and implementation levels as introduced 
earlier. 
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A complex circuit is usually made up of many functional blocks. The design task 
is eased considerably if the designer identifies these blocks. Functional blocks may 
or may not be independent, but nonetheless their identification is generally critical. 
Based on our experience, a natural solutior +o the network controller problem would 
identify many functional components, including a serial-to-parallel buffer, busy bit 
logic, and a sy?tem state buffer (idle or sending). The activities that designers demon- 
strate in this process are generally best characterized by decomposition strategies. 

A second functional activity is to coordinate functional blocks. Typically, this 
takes the form of a flow chart, state machine, or signal-flow diagram. An example of 
a flowchart for the network controller is shown in Fig. 1. In general, flowcharts and 
similar structures are used by designers to explicate interactions between modules. In 
a network controller these interactions typically relate to signals between components; 
communication between components must be coordinated if overall functionality is 
to be correct. The construction of an intermediate form (IF) of the circuit such as a 
flowchart is evidence of a designer's global plan for further design. 

<Figure 1 about here> 

4.2 Transition Level 

The next step for the designer is to implement each of the functional blocks and 
to coordinate these implementations. A number of different strategies are possible. 
Very often, designers use the transformational strategy of iterative addition or refine- 
ment by picking a core functionality, implementing this functional block in some way, 
and then implementing functions around this core block until a complete design is 
obtained. 

In contrast, experienced designers may be reminded of a more complete prototype 
and exploit a block diagram representation of that prototype system. This case-based 
strategy is typically augmented by a transformational strategy of iterative refinement, 
where local changes to the prototype are made until it satisfies the required specifi- 
cations. 
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In still other cases, after constructing a global plan for the solution of a system, the 
designer may find that certain component functions require further decomposition due 
to their complexity. Component functions may be decomposed in this manner until 
functions corresponding to known physical components are produced. More generally, 
however, there is considerable room for variance in the level of abstraction at which 
decomposition stops. For example, decomposition may continue down to basic logic 
functions that can be implemented as gates, or it may stop at more complex functions 
that are implementable as physical components such as shift registers, counters, or 
multiplexers. 

4,3 The Implementation Level and Interactions between 
Levels 

Finally, at the implementation level physical components are being manipulated. 
These manipulations often introduce interactions between components that were not 
anticipated at the function and transition levels. In some cases, interactions can 
be patched immediately at the implementation level, whereby additional physical 
components are linked in to overcome an undesirable interaction. Patches may be 
deferred as well; a designer may feel that an interaction can be patched, but it is best 
to await implementation of other functional blocks. Patching is best viewed as the 
transformational problem solving strategy. In other cases, a designer may return to 
the function level, abandon all or part of a previous decomposition, and reanalyze the 
system or part of it at the function level. This is symptomatic of creative design. 

In sum, the design process may not be a single, clean iteration through the func- 
tion, transition, and implementation levels. Furthermore, we expect that more than 
one of the problem solving strategies that we discussed - decomposition, transforma- 
tional, and case-based - will be used. 

5 Characterizing Circuit Designers 

There are several questions implied by our discussion that are relevant to the char- 
acterization of circuit designers. 
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5.1 Characteristics of the Function Level 

Questions stemming from our discussion of the function level are: 

1. How much does a designer's function level description cover the system's overall 
required functionality? 

2. Does the designer organize functional units into an intermediate form, and if 
so, what is it (e.g., flow chart)? 

3. Collectively, does the functional decomposition and intermediate form indicate 
that the designer is employing a global plan to implement the circuit. 

Our experience suggests that designers producing intermediate forms tend to pro- 
duce better and quicker designs than those that do not do this sort of planning. For 
complex designs this process is likely to happen through a series of functional de- 
compositions. More experienced designers are likely to be reminded of prototypes 
that they will employ to cover one or more functional blocks. Experienced designers 
also tend to spend more time in functional planning and decomposition producing 
greater functional detail; this behavior makes the transition process quicker and more 
concise. 

In sum, we define the following features for characterizing strategic design activity 
at the function level: 

• Number of components the designer considered 

• Type of Intermediate Form (IF) generated (e.g., flowchart). 

• Number of the Components in the Intermediate Form. 

• Amount of detail the designer introduced into the Intermediate Form. 

• Global Plan. 

The amount of detail in the intermediate form can range from low, medium to 
high. The greater the amount of detail the more strategic planning the user has done 
at the function level. Global plan refers to whether the designer explicates some form 
of global strategy in generating the design. 
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5.2 Characteristics of the Transition Level 

Questions of a transitional nature include: 

1. What strategies (decomposition, transformational, case-based) does the de- 
signer use after constructing an intermediate form? 

2. If the designer employs further decomposition, then at what level of detail does 
decomposition stop? 

3. If a transformational strategy of iterative addition or refinement is used, then 
what functional component is selected as the core component? 

At this level, observations are made as to whether the designer specifies a proto- 
type design that covers a number of functional units, or starts implementation on a 
single functional block and then includes others. Note that the designer may start 
with a complete prototype, partial prototypes, a single core component, the first unit 
in a sequence, or a random component. Designers who separate out the individual 
functional blocks that make up the designed circuit during the transition level and 
explicitly define the required communication signals between the blocks are usually 
more successful in generating correct and complete designs. The modularity created 
allows the designer to simulate and evaluate the current circuit with much less effort 
than a single large circuit. 

The following characteristics are defined to analyze behavior at the transition 
level: 

• Did the designer pick on a core component for the design? 

• Did the designer use case-based reasoning and prototypes? 

o How much detail did the designer incorporate during decomposition? 

• What kind of signal definitions did they generate to connect individual func- 
tional blocks? 
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5-3 Characteristics of the Implementation Level 

Questions at the implementation level include: 

1. What complexity of physical components does the designer use to implement 
functional blocks? These may be high-level components, such as registers, or 
low-level gates. In some cases, a designer assumes custom components, which 
cannot be taken 'off-the-shelf, but which nonetheless are easily implementable. 

2. Does the designer check for problematic interactions concerning timing of signal 
interactions between components, and if so, are these checks made locally within 
a single functional block, or globally across functional blocks? 

3. How does the designer correct undesirable interactions? Does the designer patch 
problems immediately, or does the designer defer patching, partially or entirely 
until other blocks are implemented? Does the designer simply abandon an 
evolving design in the face of complications? 

A primary characteristic here relates to the specific strategy employed by the 
designer in generating the implementation. Choice of higher level prototype blocks 
(e.g., multiplexor) and custom components (9-bit shift register) that simplify design 
are indicative of higher levels of expertise. Iterative refinement and improving the 
design implementation by making local patches with global verification is better than 
employing iterative additions which implies implementation one block at a time and 
a complete evaluation of the emerging design with the addition of each block. 

Partial checking and postponement of refinement to account for later interactions 
amongst functional blocks is indicative of higher levels of expertise. This produces 
greater efficiency because a designed block does not have to be repeatedly redesigned 
after new dependencies are discovered during implementation. 

The following summarize the features used for analyzing behavior at the imple- 
mentation level: 

• The strategy employed, 

• Use of high level components, 



• Use of custom components, 

• Interaction checks between modules, and 



• Method of problem resolution. 

6 Experimental Study 

We administered the network controller problem to eleven subjects. Three were 
considered experts. One. is a faculty member at Vanderbilt University, but he has 
designed digital systems for industrial applications (S8). The second expert works in 
the communications technology industry (S7). The third expert is a graduate student 
with extensive CMOS digital circuit design in industry (Sll). Two subjects, S4 and 
S10, had related industrial experience, but neither could be considered to be experts 
in complex circuit design. Four subjects (S2, S5, S6, and S9) are graduate students 
with little or no professional design experience. The other two subjects were seniors 
in our undergraduate program at Vanderbilt university. They both had completed 
a senior level course in digital circuit design. However, subject S3 was regarded as 
a high performing student, whereas Si was considered to be average in academic 
performance. 

6.1 Protocol Analysis 

The subjects were given a hardcopy of the problem description (see Appendix A), 
and asked to develop their solution with pencil and paper. They were requested to 
think aloud as they developed their solution, and the entire session was videotaped. 
The subjects could use any material (notes, books, etc.) that they felt would assist 
them in generating a solution. A complete transcript of each session was generated. 

Later a group of raters (Biswas, Glewwe, Bhuva) studied the tapes, transcripts, 
and the subjects' written output. To encode strategic activity, we used the following 
two-step process to analyze subjects' data: 

1. In Step 1, we encoded the implementation- level blocks created by the sub- 
jects, and the sequence in which they added, deleted, and modified blocks as 
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they went through the design process. In other words, this trace records the 
designer's activities at the implementation level. Most implementation level 
blocks originated from a function-level description and retain the same name as 
the functional block (e.g., serial-to-parallel buffer). Subjects usually put down 
this name when they drew this block as part of the implementation on paper, 
or they verbally expressed the name of the block as they drew it on paper. 

2. In Step 2, we used the Step 1 trace and information from the tapes and tran- 
scripts to answer the set of questions that define our framework for character- 
izing design activity (discussed in Section 4). This provides information that 
describes strategic level behavior of subjects at the function, transition, and 
implementation levels. 

The trace generated in Step 1, in addition to providing information for creating 
Step 2, also helps us check the correctness of the design. If the subject has errors in 
the design, the trace helps us understand where and how a subject went wrong. In 
combination with the Step 1 trace, it helps determine the subject's characteristics, 
such as (a) did the subject attempt to pick up higher level blocks and off-the-shelf 
components to simplify the design, (b) did the subject implement more at the gate 
level, (c) if so, did the subject try and optimize the design in terms of the number of 
gates, and (d) did the subject think about chip design in component selection. 

The Step 2 trace is the key to recognizing levels of expertise. The sequence of 
reasoning methods applied at the function, transition, and implementation levels is a 
key indicator. For example, two designers may produce the same final solution. The 
first designer may employ case-based reasoning methods to generate a complete proto- 
type system, while a second designer may not have access to a ready-made prototype 
model. Therefore, this designer uses function-level description to understand what is 
required, and then completes his/her design primarily from first principles. The first 
designer's Step 2 trace will show function-level activity, followed by transition-level 
activity, and then implementation-level activity. The second designer's Step 2 trace 
is likely to demonstrate back-and-forth activity between the function, transition, and 
implementation levels. Circuit implementation is likely to occur more by incremental 
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additions. 



6.2 Process Characteristics 

The dimensions described in Section 4 were applied to analyze the Step 2 behavioral 
characterization of the 11 subjects. Tables 1-3 summarize the individual behaviors 
along the function, transition, and implementation levels, respectively. 



<Table 1 about here> 
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Table 1 shows the evaluation of each subject at the function level. For this analysis, 
we assumed a normative solution, consisting of 7 functional blocks, and an interme- 
diate form given by the flowchart in Figure 1. This constitutes the 'gold standard' 
against which we compared subject solutions. We were interested in assessing how 
close to this solution each subject came. Because subjects might differ in the number 
and organization of functional blocks from the norm, we developed an algorithm for 
counting the amount of 'functional' overlap. If a designer defines a functional block 
that covers more than one of the 7 specified blocks, all 'covered' blocks in the norma- 
tive solution are counted as considered. If the designer uses finer-grain blocks than 
the ones specified, a set of finer-grain blocks are counted as a larger normative block 
if the designer effectively 'implements' this block with appropriate links between the 
finer grained blocks. For example, subject S3's functional description covered 6 blocks 
of the normative solution. 

In addition, Table 1 differentiates between the number of functional blocks ini- 
tially considered (obtained from recorded protocols), and the number of blocks that 
were represented in the final intermediate form; these are given in columns 1 and 3, 
respectively. Other features used for assessment are the type of Intermediate Form 
(IF) used, details specified in IF (i.e., detail in specifying links between blocks), and 
verbal evidence for the presence of a global plan (i.e., some explication by the subjects 
of the steps that they will follow in fleshing out the design). Some of the subjects did 
not use IF and they were evaluated based on the rest of the design features. The use 
of higher number of components, greater detail in IF, higher number of components 
in IF, and the presence of a global plan imply higher expertise. However, in assigning 
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an overall rating of observed competence at the function level (column 6), we focused 
on the coverage and detail of the intermediate form: no IF (rating 1), low/medium 
coverage of IF (relative to normative solution) and low/medium detail (2), at least 
medium coverage and detail (3), and both high coverage and detail (4). 

<Table 2 about here> 

Table 2 shows the evaluations of each subject at the transition level. The features 
used are the details in decomposition and the use of interaction signals during this 
level. Both of these measures are relative measures. In general, the more fine-grained 
the decomposition and the more detailed the anticipation of interactions through 
signals, then the greater the ease with which design should proceed and the greater 
the presumed level of expertise. Column 4 lists overall ratings obtained as follows: 
low decomposition and use of signals (1), at least medium decomposition and no more 
than medium use of signals (2), high decomposition or use of signals, but not both 
(3), both high decomposition and high use of signals (4). 

<Table 3 about here> 

Table 3 shows the evaluations of each subject at the implementation level. The 
features used for assessment are the use of higher-level components, use of custom 
components, types of interaction checks, and the strategy employed in problem res- 
olution. This table does not give an overall rating, but as we noted earlier, the use 
of high level and custom components demonstrates greater expertise, as does global 
versus local interaction checks. In addition, most subjects used patching to correct 
for interactions. In contrast, subject Si abandoned a partially flawed design, which 
is indicative of low expertise. 

6.3 Product Characteristics 

In addition to characterizing subjects in terms of process characteristics, we can 
also characterize them in terms of the completeness and correctness of their final 
designs. Table 4 gives a qualitative score for the completeness of each subject's final 
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implementation (i.e., what proportion of the specified functionality did the final design 
cover) and a score for correctness (i.e., the proportion of specified subfunctionalities 
covered by the final implementation). Table 4 also classifies each subject on a four 
point scale, on the basis of the combined completeness and correctness ratings. Level 

1 scores reflect -inimal completeness and correctness in final implementations. Level 

2 scores reflect radium completeness and correctness. Level 3 subjects scored high on 
one dimension and medium on the other. One subject was classified at Level 4, and 
scored high on both dimensions of completeness and correctness. We have left subject 
S7 unclassified, because he took a radically different approach to the controller design 
than other subjects. S7 employed a programmable logic array in solving the problem, 
and was stopped early by the experimenter prior to full implementation. 

<Table 4 about here> 

6.4 Integrating Process and Product Characteristics 

The product rating and the previously-described process characteristics are generally 
consistent with one another. An exception to this was S3 who performed well at the 
function and transition levels. This exception would have gone unnoticed had we 
only looked at product scores. We were concerned that other important differences in 
design activity were being masked by focusing on the product measure. We developed 
a more integrated rating scheme consisting of four levels as follows. 

1. Novices show little proficiency in complex des ; 3n. Novices scored low in com- 
pleteness and correctness, and had low functional and transitional scores as 
well. 

2. Average designers scored medium for completeness and correctness, or per- 
formed well at the functional and transitional level, which is indicative of a 
sound understanding of the problem and a strategy for proceeding. 

3. Above average designers demonstrate good understanding of complex design, 
and the ability to see the design process through to an adequate solution. Their 
average completeness and correctness was above medium. 

15 
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4. Expert designers have a very good understanding of design. In our study we 
classified one subject as expert. This individual scored high on both dimensions 
of completeness and correctness, and demonstrated mastery at the functional 
and transitional levels. 

These categories are exemplified in the descriptions of the design problem solving 
behavior of our subjects, as follows. 

6.4.1 Novice Designers 

SI and S2 were ranked as novices. Si understood some basic principles of digital 
design, but did no function-level design. Si could implement parts of the circuit, 
but showed complete lack of understanding of others. He made no attempt to de- 
compose the circuit before implementation. Instead, he tried to iteratively add new 
components, but could not handle flaws that arose because of interactions. S2 was 
more proficient in function-level design, but had no idea how to convert the functional 
blocks into a circuit implementation. His main problems were in transition-level de- 
sign. He did not use signals to separate functional blocks of the circuit and became 
bogged down with trivial details. Both subjects might be regarded as performing 
creative design: failure to adequately decompose and isolate functionality led to non- 
determinism at lower levels, which were difficult for these subjects to handle. 

6.4.2 Average Designers 

S3-S5 were ranked average designers. S3 and S* did some function-level design but 
none of the subjects had a complete understanding of the problem, therefore, they 
had difficulty in transition. All three did break down the system into some modules 
(incomplete) and were able to define signals to connect these modules. S5 did not 
include a portion of the circuit because he misunderstood the problem and was not 
corrected by the experimenter. S3 could not complete the implementation, as the two 
N/A entries in Table 3 might suggest, but nonetheless demonstrated proficiency at 
the function and design levels of the design process. Note that if we had only taken 
into account completeness and correctness aspects of S3's final design, then S3 would 
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have been classified as a novice. S4's background in analog circuit design oriented him 
to focus on signal generation and timing analysis, but this approach did not produce 
a working implementation. He had more difficulty than the others assimilating the 
problem. 

6.4.3 Above- Average Designers 

Five subjects, S6-S10, were placed in this category. Three of the subjects produced 
good function-level design and all of them generated good solutions. S9 and SlO 
ignored function-level design and selected a shift register as a ring data buffer, re- 
spectively to start off the design process. This was notable, but isolated evidence 
of case-based reasoning. They did not have complete prototypes, however, and used 
iterative refinement to add to these components and complete their design. SlO used 
signal analysis to keep track of his overall design. Overall, barring the differences 
discussed, S6, S8, S9, and SlO's approach were similar, and their solutions were com- 
parable. In contrast, S7 produced a non-standard solution and was stopped early by 
the experimenter prior to full implementation. Although his solution was not com- 
pletely specified, he seemed to know exactly what he was doing and probably would 
have generated the complete solution had he been forced too. We can safely say that 
S7 was at least an above average designer, and might well have qualified as an expert 
had he finished the problem. 

6.4.4 Expert Designers 

Sll undoubtedly had a superior approach and produced the best solution to the 
problem. He performed detailed function-level design, and worked on refining the 
solution at this level before transitioning to the implementation level. Most of his 
analysis and checking occurred at the function level, therefore, error correction and 
recovery proved to be much easier for him than the other subjects who did a lot of 
evaluation and checking at the implementation level. The relative determinism at 
each stage suggests that Sll came the closest of our subjects to performing routine 
design. 
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6*5 Implications for Assessment of Expertise 

We noted at the outset that one issue for purposes of assessment is whether both 
process and product characterizations are necessary indicators of expertise or are 
important in making expertise classifications. It seems clear to us that there is im- 
portant information to be gained by looking at how people went about producing a 
final design. For example, having only looked at product we would not have known 
that subjects 9 and 10 used a primitive form of case-based reasoning and did not use 
functional decomposition. Likewise, subject 3's performance was due to difficulties at 
the implementation level, while her performance at the function and transition levels 
was quite good. We would also not have known that subject 4 attempted to transfer 
his expertise in analog circuit design to the digital case. Furthermore, subject 11 's 
behavior suggests the importance of early error checking and evaluation at the func- 
tional level in obtaining a good design. The information gained from the process level 
descriptions provides a basis for comparisons with other domains in which expertise 
has been described. In addition, a comparison of the processes used by the most 
expert subject in our group with the processes of the less expert subjects provides a 
basis for instructional interventions. On the other hand, it is also clear that there are 
circumstances under which the product characteristics are sufficient for classification. 

7 Ongoing Work on an Automated Design Tool 

Assuming that one is interested in the richer characterization that results from con- 
sidering both process and product in digital circuit design, it is important to make 
the collection of process data more tractable than through the analysis of think-aloud 
verbal protocols. Towards this, end, we are building an automated design tool that 
can be used by subjects for purposes of assessment. The design tool has the following 
components: 

1. problem-description browsing tool 

2. high-level formalization and simulation tool 

3. circuit- diagram drawing tool, and 
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4. circuit-simulator tool. 

We are currently studying issues related to the high-level formalization and circuit- 
diagram drawing tool. 

1. How natural is the high-level formalization tool for creating an intermediate 
representation of the design problem. More specifically, we are studying how 
easy it is for subjects to express their' high-level functional conceptualizations 
in a flow-chart like form. A side issue we are trying to capture is how many 
levels of decomposition they produce at the function level. 

2. Use of the circuit diagram drawing tool. We noticed that subjects went through 
a number of iterations when creating their Implementation level designs. They 
added blocks to existing blocks, changed and refined blocks and often got rid 
of blocks and redesigned them. Sometimes they had to move them around. On 
paper most of these tasks required them to redraw figures, which is a nuisance. 
Also, but for the full protocols it would be hard to recognize what blocks were 
being deleted and replaced, or refined. However, the computer tool lets them 
save intermediate implementations for future reference and also to import past 
representations onto the current drawing area and modify them. This should 
make it much easier to capture and interpret the activities discussed above. 

A preliminary version of this tool, was evaluated within the group and with a 
few subjects during development. The general consensus was that this tool was too 
limiting in its ability as a CAD tool. Features in the high level functional tools and 
the circuit implementation tool were not extensive enough. It was determined that a 
framework for combining multiple design assistant tools was needed. Designers must 
be able to spend almost all of their energy on design, not on the tool's workings. 

We believe some important requirements for a usable assessment tool are: 

• Designers must be able to guide the design process at all times. Currently, 
most tools center around complete synthesis of a circuit from a very formal 
user-specified design language such as a VHDL, or the tools force the designer 
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to specify all of the low-level design (just a drawing tool). Something between 
these two is necessary; a tool that can handle tedious details of synthesis but 
still allows the designer to intervene and force specific choices. 

• Tools must be developed for working completely at the function design level. 
These tools need to handle flowchart creation or state machine creation by the 
user in a graphical format. A graphical format is necessary for the designer to 
more easily grasp what is being entered. Completely textual system'- describe 
circuits such that they are difficult to understand. 

• Simulators for functional tools are necessary that can directly simulate the 
graphical representations of flowcharts and state machines. This implies that 
the functional descriptions will have to be somewhat formal. 

While these components are certainly the most important to us, the framework 
built for integrating these tools should be expandable so new tools can be added as 
needed. 

8 Conclusions 

This study has suggested a more complete and rich process model for capturing 
behavior in complex design. We have proposed characterizing CMOS circuit design 
behavior at different levels of abstraction - function, transition, and implementation 
levels. We believe that this provides a richer language for classifying designers into 
different levels of expertise. Empirical studies demonstrate considerable variance 
along these levels over subjects with differing design abilities. 

We are currently working toward developing a computer-based tool for data gath- 
ering and assessment of design skills. Once this tool is complete, it will be important 
to study the reliability of computer-generated data in classifying design behavior as 
think-aloud protocols and paper-andi-pencil problem solving. Lastly, we need to ex- 
tend our study and perform a more systematic analysis of strategic and lower (i.e., 
unit operator) reasoning levels in order to derive a more formal model for assessment. 
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Appendix A 

A small company selling networking supplies is experiencing problems with one of 
their IC suppliers and decides to design their own network controller. The type of 
network the company sells is a proprietary token ring architecture. The token ring 
controller will be manufactured as a CMOS chip by an external firm, using 2 micron 
technology. The network normally connects 5 to 100 computers, with each computer 
containing one network controller. You are asked to design the sender part of the 
controller. The final design should be a gate level design specification. You are free 
to use higher level modules like shift registers, just specify the implementation of one 
bit of the register at the gate level. / 

<Figure Al about here> 

The computers are connected in a ring topology. Each computer has a unique 
7 bit address. A frame of data is of variable length and consists of the following 
elements: 

1. an 8-bit token, indicating the head of a frame (1000 0000) 

2. a 1-bit busy signal, indicating whether frame is full or empty (1 = full, 0 = 
empty) 

3. a 7-bit address identifying the intended receiver 

4. a 1-bit signal acknowledging reception by the receiver (1 = accepter, 0 = other) 

5. a 7-bit address identifying the sender 

6. an N-byte data packet 

<Figure A2 about here> 
The controller has the following lines: 
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1. (RI) Ring in (input) Connected to the ring (receiving data) 

2. (RO) Ring out (output) Connected to the ring (sending data) 

3. (DI) Data in (input) Connected to the system; Input for data to be sent 

4. (DF) Data finished (input) System has no more data 

5. (FD) Fetch data (output) Controller needs next data bit from system 

6. (CL) Clock (input) Synchronous signal for all computers 

<Figure A3 about here> 

When the controller receives the token, it checks the next bit to see if the following 
frame contains data. If it does: the controller does nothing (receiver part checks to 
see if data is intended for this system, and initiates reception). If the frame is empty 
the local system is allowed to send data if it has any available (DF = 0). In that 
case the next 7 bits are set to the address of the intended receiver. The following 
bit is set to 0 (acknowledge signal), and the next 7 bits are set to the address of the 
sender. Finally data transmission begins. When no more data is available (DF = 1) 
transmission stops. While the system is sending it keeps watching for the return of the 
token. When it receives a busy bit following return of the token (busy bit generated 
by itself) it HAS TO stop sending and reverse the busy bit to 0. Generation of sender 
and receiver addresses all happens externally on the Data In (DI) line. NOTE: You 
are responsible for two outputs (Ring Out and Fetch Data) derived from four inputs 
(Ring In, Data In, Data Finished, and Clock); see the figure on the previous page for 
explanations of these signals. 

Additional Constraints: 

• Minimum buffer size per controller: 8 bits 

• Maximum buffer size per controller: 16 bits 

• Speed of operation: 10 Mbps 

Please do not worry about the following aspects: 

• generation and synchronization of clock signals 

• power down situations in any of the computers on the ring 
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conflicts between data and marker bit patterns 
reception of packets 
loss of token 

no acknowledgement from receiver 
limited buffer capacity of the net 
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Table 1: Function level evaluation. 



Number of 
Components 



IF 



Number in 
IF 



Detail in 
IF 



none 



n/a 



Rough State Machine 



n/a 
Low 



Flowchart 



High 



Global 
Plan 
No 
No 
No 



none 



n/a 



n/a 



No 



Flowchart 



Low 
Medium 



No 



Flowchart 



No 
No 



Data Flow Diagram 



Medium 
Medium 



Informal State Machine 



Yes 



none 



n/a 



n/a 



No 



none 



n/a 



State Machine 



n/a 
"High" 



Yes 
Yes 



SI 



Table 2: Transition level evaluation 



Core Component 



Switch 



Detail in 
Decomposition 



High 



Use of 
Signals 



Low 



Score 



_2_ 
1 



S2 



Shift Register 



Low 



Low 



S3 



S4 



Abstract Controller 



High 



Shift Register 



Medium 



Medium 



Medium 



S5 



S6 



S7 



System State Flop 



High 



Shift Register 



High 



ROM Sequencer 



Medium 



Low 



S8 



Shift Register 



Medium 



Medium 



S9 



S10 



Sll 



Shift Register 



Medium 



Shift Register 



Medium 



High Level Blocks 



High 



Low 



High 



High 
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Table 3: Impleinentational Level 



Strategy 



High Level Custom 



Interaction 



Problem 



Iterative Addition 


Yes 


No 


Local Some 

yJlODal 


nesomnorc 

Abandon Flawed 
& Correct Parts 


Iterative ReSne. & 
Addition 


Yes 


Yes 


Local (very 

UUiC ) 


Immediate Patch 


n/a 


Yes 


Yes 


none 


n/a 


Iterative Ratine. & 
Addition 


Yes 


No 


Local 


Immediate Patch 


Iterative Retine. & 
Addition 


Yes 


No 


Global 


Immediate Patch 


Iterative Addition 


Yes 


No 


Local 


Immediate Patch 


Complete Block 
Layout 


Yes 


No 


none 


n/a 


Global Layout; 
Iterative ReSne. 


Yes 


Yes 


Local w/ a 
Final Global 


Deferment 


Iterative Retine. &. 
Addition 


Yes 


Yes 


Local 


Immediate Patch 


Iterative Retine. & 
Addition 


Yes 


Yes 


Local 


Immediate Patch 


Iterative Ratine, on 
Signals/3 locks 


Yes 


Yes 


At Block 
Level 


Immediate Patch 
fOne Deferment) 



9 

ERIC 



37 



Table 4: Evaluation 





Completeness 


Flaw 
Detection 


Overall 
Rating 


SI 


Low 


Low 1 




1 


S2 


Low 


Low 




1 


S3 


Low 


none 




1 


S4 


Medium 


Medium 




2 


S3 


Medium. 


Medium 




2 


S6 


High 


Medium 




3 


S7 


Medium 


n/a 






S8 


Medium 


High 




3 


S9 


Hi2h 


Medium 




t 3 


S10 


Hism 


Medium 




3 


SU 


Ksm 


High 




1 4 



o 
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(Sendar Part of Controller) 




Figure 1: Flowchart form: Token Ring Controller 
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